Papers with pretrained embeddings
Wikipedia2Vec: An Efficient Toolkit for Learning and Visualizing the Embeddings of Words and Entities from Wikipedia (2020.emnlp-demos)
Copied to clipboard
Ikuya Yamada, Akari Asai, Jin Sakuma, Hiroyuki Shindo, Hideaki Takeda, Yoshiyasu Takefuji, Yuji Matsumoto
| Challenge: | Existing tools for learning the embeddings of words and entities from Wikipedia are not yet available. |
| Approach: | They propose a Python-based tool for learning Wikipedia embeddings from Wikipedia . they use a Wikipedia dump file as an argument to issue a single command . |
| Outcome: | The proposed tool achieves state-of-the-art results on the KORE entity relatedness dataset and competitive results on benchmark datasets. |
Sequence Tagging with Contextual and Non-Contextual Subword Representations: A Multilingual Evaluation (P19-1)
Copied to clipboard
| Challenge: | Pretrained contextual and non-contextual subword embeddings are available in over 250 languages, allowing massively multilingual NLP. |
| Approach: | They compare pretrained contextual and non-contextual subword embeddings with a contextual representation method, namely BERT, on multilingual named entity recognition and part-of-speech tagging. |
| Outcome: | The proposed method outperforms non-contextual embeddings on multilingual named entity recognition and part-of-speech tagging. |
Poetry to Prose Conversion in Sanskrit as a Linearisation Task: A Case for Low-Resource Languages (P19-1)
Copied to clipboard
| Challenge: | Obtaining the proper word ordering, called as the prose ordering, from a verse is often considered a task which requires linguistic expertise. |
| Approach: | They propose a word ordering (linearisation) task that ignores the word arrangement at the verse side. |
| Outcome: | The proposed model outperforms current models in word ordering for the translation task in Sanskrit. |
“Chère maison” or “maison chère”? Transformer-based prediction of adjective placement in French (2023.findings-eacl)
Copied to clipboard
| Challenge: | Prior work has shown that transformer-based language models are insensitive to permutated word order, but this is not the case with pretraining. |
| Approach: | They conduct experiments to assess whether transformer-based language models are able to learn the adjective position in noun phrases in French. |
| Outcome: | The proposed model is weaker with complex structures and fixed expressions, but favors context and global syntactic roles. |
Modular Monolingual Adaptation using Pretrained Language Models (2026.acl-industry)
Copied to clipboard
| Challenge: | Existing approaches to building monolingual models for low-resource languages require a full model tuning process. |
| Approach: | They propose a modular approach to build monolingual models for low-resource languages by finetuning the whole model on the target language. |
| Outcome: | The proposed model improves on natural language understanding tasks on Scottish Gaelic, Irish, and Quechua with Quechuan being a very low-resource language. |
Topic Sensitive Attention on Generic Corpora Corrects Sense Bias in Pretrained Embeddings (P19-1)
Copied to clipboard
| Challenge: | Existing methods to adapt pretrained embeddings to a large corpus are limited and do not provide sufficient quality. |
| Approach: | They propose to use a small corpus D_T to pretrain embeddings that accurately capture the sense of words in a limited set of focused topics. |
| Outcome: | The proposed embeddings capture the sense of words in a topic in spite of the limited size of the corpus D_T. |
Contextual Embeddings: When Are They Worth It? (2020.acl-main)
Copied to clipboard
| Challenge: | In recent years, rich contextual embeddings have enabled rapid progress on benchmarks like GLUE, but require significant computational resources during pretraining and during downstream task training and inference. |
| Approach: | They empirically compare contextual embeddings with classic pretrained embedders and a random word embeddable with a simple baseline. |
| Outcome: | The proposed models perform within 5 to 10% accuracy on industry-scale data. |
IndiSocialFT: Multilingual Word Representation for Indian languages in code-mixed environment (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on word embeddings for Indian languages focus on monolingual corpora with limited reach to social media setups. |
| Approach: | They propose a generalized representation vector for diverse text characteristics . they use a FastText model to gather text from social media and well-formed sources . |
| Outcome: | The proposed representation vector surpasses baselines in most cases and languages, demonstrating suitability for various NLP applications. |
Trial2Vec: Zero-Shot Clinical Trial Document Similarity Search using Self-Supervision (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Clinical trials are expensive and time-consuming to conduct, and lengthy trial documents and lack of labeled data make comparisons difficult. |
| Approach: | They propose a zero-shot clinical trial retrieval method which learns through self-supervision without the need for annotating similar clinical trials. |
| Outcome: | The proposed method improves on baselines on precision/recall and 15% on the downstream trial outcome prediction task. |
Cooking Up a Neural-based Model for Recipe Classification (2020.lrec-1)
Copied to clipboard
Elham Mohammadi, Nada Naji, Louis Marceau, Marc Queudot, Eric Charton, Leila Kosseim, Marie-Jean Meurs
| Challenge: | a dataset of cooking recipes in French is highly imbalanced due to collaborative nature of the dataset . authors propose a neural-based model to address the first task of the DEFT 2013 shared task . |
| Approach: | They propose a neural-based model to address the first task of the DEFT 2013 shared task . they use state-of-the-art embedding approaches and deep architectures to address imbalanced dataset . |
| Outcome: | The proposed model outperforms models that use only pretrained embeddings in micro and macro F1 scores. |